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ABSTRACT 


tee  of  the  ways  of  assuring  greater  reliability  of  software  is  to 
improve  testing  techniques.  Three  of  the  key  problems  associated  with 
software  testing  are:  choosing  adequate  test  cases,  assuring 

correctness  of  the  results,  and  reducing  the  high  cost  of  testing.  Sene 
degree  of  automation  is  required  to  help  solve  these  problems.  By 
combining  the  capability  of  adaptive  testing  with  the  use  of  executable 
assertions,  it  is  possible  to  automatically  execute  a  program  with  i 
large  number  of  testcases  over  a  wide  range  of  input  values.  The 
usual  goal  of  adaptive  testing  is  to  maximize  some  performance  value 
(objective  function)  for  the  software  by  automated  perturbation  of  the 
input  parameters  in  such  a  way  as  to  degrade  the  system  performance  to  a 
specified  limit.  This  technique  only  indirectly  leads  to  locating 
errors,  because  of  the  time- consuming  task  of  examining  the  usual  output 
from  the  program.  In  software  testing,  the  primary  goal  is  to  locate 
the  maximus  number  of  errors  rather  than  maximize  the  performance  value. 
Since  software  errors  oan  be  deteoted  by  executable  assertions,  these 
assertions  can  be  used  to  define  an  objective  function  for  the  adaptive 
tester  so  that  a  program  can  be  tested  automatically  and  a  mapping  made 
of  its  "error  spaoe."  A  search  algorithm  is  used  to  generate  new  test 
cases  based  on  past  performance  data  about  the  number  of  assertion 
violations.  Software  testing  can  become  much  more  efficient  and 
effective  through  the  use  of  adaptive  testing  with  assertions,  because 
suoh  extensive  testing  increases  the  possibility  of  finding  any  existing 
errors  and  of  improving  software  reliability. 
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Testing  is  one  area  of  the  software  development  cyole  where  there 
is  a  need  for  vast  improvement.  It  is  frequently  the  most  costly  part 
of  the  cycle  and  the  most  time-consuming.  What  is  needed  is  not  to  put 
more  money  and  time  into  testing,  but  to  devise  a  systematic  method  of 
exercising  a  program  with  a  sufficient  number  of  input  values  over  the 
entire  range  of  possible  values  in  such  a  way  that  any  existing  errors 
are  discovered.  To  accomplish  this  goal  and  to  automate  as  much  of  the 
testing  process  as  possible,  the  techniques  developed  for  adaptive 
testing  are  being  combined  with  the  use  of  executable  assertions  for 
error  detection. 


Before  software  testing  can  be  automated  (and  become  cost 
effective),  two  primary  problems  must  be  solved:  developing  adequate 
test  cases  to  identify  errors,  and  verifying  the  results  of  these  test 
cases.  The  problem  of  developing  test  cases  has  been  studied  in  the 
Adaptive  Verification  and  Validation  research  program.1  The  method  has 
been  to  use  various  search  techniques  adapted  from  optimization  theory 
and  artificial  intelligence  research  in  order  to  maximize  a  performance 
value  (objective  function  for  the  software).  The  original  test  cases 
(input  values)  are  either  supplied  by  the  tester  or  developed 
stochastically.  These  values  are  then  altered,  through  a  feedback 
mechanism  using  heuristios,  to  maximize  the  performance  value. 


In  software  testing,  however,  we  are  not  so  much  interested  in 
maximizing  performance  as  in  locating  errors.  The  technique  of 
maximizing  a  performance  function  only  indirectly  helps  to  locate  errors 
in  the  software.  Errors  are  usually  discovered  by  examining  the  output 
of  a  program,  whioh  is  a  time-cons  using  task.  Since  software  errors  can 
be  detected  through  the  use  of  assertions,  assertions  can  be  used  as  the 
objective  function,  thus  allowing  the  adaptive  testiig  techniques  to  be 
applied  to  the  problem  of  testing  software. 


ADAPTIVE  TESTING 

Adaptive  testing  is  a  technique  for  Identifying  how  well  a  prograa 
per fonts  in  response  to  changes  in  its  input  values.  The  Adaptive 
Tester  was  developed  to  test  the  response  of  simulated  ballistic  missile 
defense  programs  to  changes  in  a  threat  scenario.  The  program's 
performance  was  defined  by  the  number  of  re-entry  vehicles  which  were 
not  Intercepted. 

The  Adaptive  Tester  uses  the  principles  of  feedback  and  adaptive 
searoh  to  identify  scenarios  which  result  in  the  maximtn  tolerable 
number  of  re-entry  vehicles  penetrating  the  defense.  An  initial 
scenario  is  described  and  input  to  the  test  case  construction  algorithm. 
This  algorithm  generates  a  set  of  input  values  for  the  program  being 
tested.  The  program  is  executed  and  values  are  recorded  which  measure 
its  performance.  The  performance  values  are  then  evaluated  and  compared 
with  past  performance  values.  The  change  in  performance  values  is  then 
used  as  input  to  the  adaptive  search  algorithm  which  constructs  a  new 
soenario.  New  input  data  is  constructed  for  this  scenario  and  the 
program  is  run  again.  This  cycle  continues  until  scenarios  are  found 
which  cause  the  maximum  tolerable  number  of  re-entry  vehicles  to 
penetrate  the  defense.  The  input  values  which  characterize  each  such 
scenario,  and  the  resulting  performance  values,  define  the  "performance 
boundary"  of  the  program. 

MBIliQfiQLQfiX  FQfl  ADAPTIVE  lESIIfltt  KIIH  ASSfiflIIQBS 

In  much  the  same  way  as  the  input  parameters  are  systematically 
perturbed  to  degrade  the  system  performance  until  the  performance 
boundary  is  reached,  the  input  parameters  of  a  program  can  be  perturbed 
until  areas  with  maxima  errors  (indicated  by  assertion  violations)  are 
located.  The  first  step,  therefore,  is  to  add  assertions  to  the  code  to 
be  tested,  if  that  has  not  already  been  done.  Actually  assertions  have 
many  uses  and  are  extremely  valuable  throughout  the  entire  software 
cycle.  Ideally  they  should  be  written  during  the  design  phase  to  state 
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specifications  about  the  variables  before  any  coding  takes  place. 
Later,  during  dynamic  tests,  these  same  assertions  can  be  made 
executable  to  help  in  program  debugging.  Assertions  can  also  be  left  in 
the  code  as  a  form  of  docuaentation,  because  they  can  contain  much 
useful  information  about  variables  (e.g.,  the  expected  range  of  values 
or,  in  distributed  systems,  path  or  timing  constraints).  These  built-in 
specifications  are  a  way  of  protects  the  software  during  deploynent  or 
maintenance  from  modifications  to  the  code  which  may  alter  the  expected 
mode  of  operations. 

Almost  any  condition  or  specification  can  be  expressed  using 
executable  assertions.  An  executable  assertion  is  a  logical  expression 
which,  if  evaluated  to  false,  signals  the  violation  of  a  specification 
for  the  program.  The  logical  operators  of  the  assertions  have  been 
extended  to  include  the  operators  of  first-order  predicate  calculus: 
implication,  existence,  and  universal  quantifiers.  A  preprocessor 
translates  the  assertions  into  executable  statements.  When  the  program 
is  executed,  the  logical  expression  in  each  assertion  is  evaluated.  If 
it  is  false,  an  error  message  is  printed  which  states  the  name  of  the 
module  and  the  line  number  of  the  assertion  statement.  In  addition,  a 
tabulation  is  made  of  how  many  times  an  assertion  is  violated. 

The  important  role  of  the  assertions  in  this  type  of  testing 

cannot  be  emphasized  enough:  they  should  be  interspersed  throughout  the 
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code  at  appropriate  places  ,  there  must  be  a  sufficient  number  of 
assertions  to  monitor  each  variable,  and  they  must  correctly  state  the 
performance  requirements  of  the  variables.  In  other  words,  the  success 
of  this  type  of  testing  depends  on  the  validity  and  comprehensiveness  of 
tbs  assertions. 

To  assure  the  correctness  of  the  assertions,  the  second  step  is  to 
perform  preliminary  tests  of  the  program  with  varied  input  values.  A 


frequent  result  of  this  initial  testing  is  that  unsuspected  errors  in 
the  code  are  uncovered  by  the  assertions.  Any  errors  in  the  code  or  the 
assertions  should  be  corrected  before  continuing. 


The  third  step  is  to  construct  a  set  of  test  data  by  specifying 
the  possible  range  of  values  for  the  input  variables.  The  initial  value 
of  a  variable  is  the  minimum  boundary  value  and  the  upper  limit  (and 
final  value)  is  the  maximum  specified  value.  Next  some  interval  is 
chosen  through  which  the  value  of  the  variable  will  be  stepped  during 
the  testing.  In  this  way,  a  "grid"  of  input  values  is  defined  over  the 
input  space.  Figure  1  shows  the  input  space  boundaries  for  a  program 
which  was  used  as  part  of  an  experiment  to  determine  if  it  was  feasible 
to  merge  the  adaptive  testing  methods  with  the  use  of  executable 
assertions  for  this  type  of  testing.  The  input  values  of  the  transit 
times  were  within  the  range  of  0  to  300  microseconds.  The  probability 
of  any  one  pulse  request  in  the  sequence  of  requests  being  a  search 
pulse  was  allowed  to  range  between  0  and  1. 


NIMER  OF  ASSERTIONS  VIOLATED 


TRANSIT  TINE 
(NICROSECONDS) 


PROPORTION  OF  SEARCH  PULSES 
(PRORARILITY) 


Figure  1.  Input  Space  Boundary 


Once  the  range  of  input  values  is  specified  and  the  interval 
chosen,  the  rest  of  the  testing  is  automated.  The  tester  is  relieved 
not  only  of  the  ohore  of  choosing  new  input  data  but  also  of  the  tedious 
task  of  wading  through  reams  of  output  to  verify  the  results  of  each 
test  case.  During  the  fourth  step,  the  program  is  executed  once  for 
each  set  of  input  values  and  a  mapping  is  made  of  the  error  space.  The 
input  variables  of  the  program  are  defined  as  the  independent  variables, 
and  the  number  of  assertions  which  became  false  when  the  program  is  run 
with  a  particular  set  of  input  values  is  defined  as  the  dependent 
variable.  Values  of  the  dependent  variable  define  an  objective  function 
of  errors  over  the  input  space.  By  maximizing  this  objective  function, 
we  can  locate  the  values  of  the  input  variables  which  cause  the  program 
to  fell . 

Having  defined  the  error  function  in  much  the  same  way  as  the 
performance  function  is  defined  for  the  Adaptive  Tester,  the  fifth  step 
then  is  to  use  the  search  algorithms  of  the  Adaptive  Tester  to  locate 
values  in  the  program's  input  variables  which  cause  the  most  errors  to 
occur.  The  reason  we  search  for  the  area  of  maximum  assertion 
violations  is  to  uncover  as  many  errors  as  possible.  Although  some 
errors  will  cause  several  assertions  to  be  violated,  other  errors  are 
only  indicated  by  a  single  assertion  failure;  therefore,  it  is  essential 
that  the  entire  input  space  be  searched  for  all  possible  violations. 

Figure  2  shows  the  organization  of  the  adaptive  testing  programs. 
A  test  case  construction  algorithm  takes  the  initial  test  data  and 
oontructs  a  set  of  input  values  with  which  to  run  the  program.  These 
input  data  values  are  also  recorded  in  the  test  results  file  for  use  by 
the  search  algorithm  in  altering  the  input  data.  The  program  is  run  and 
the  assertion  evaluator  evaluates  the  assertions  as  they  are  executed 
and  records  any  assertions  that  become  false.  The  test  results  file  is 


then  read  by  the  search  algorithm,  which  computes  new  values  for  the 
input  variables  based  upon  the  assertions  violated  and  the  past  history 
of  the  tests.  These  new  values  are  input  to  the  test  case  construction 
algorithm,  which  forms  a  new  set  of  input  values  for  the  program  and 
executes  it. 
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Figure  2.  Software  for  Adaptive  Testing  Using  Assertions 


The  process  that  generates  a  radar  schedule  in  a  missile  defense 
simulation  was  chosen  as  the  test  object  for  a  preliminary  evaluation  of 
this  method  of  testing  software.^  This  set  of  modules  take  a  sequence 
of  requests  for  radar  pulses  and  from  these  constructs  a  schedule  for 
the  radar's  time.  This  schedule  must  not  overlap  transmitted  pulses 
with  each  other  or  with  listening  times  ("receive  windows")  and  must 
allow  sufficient  time  between  pulses  and  receive  windows  to  switch  beam 
positions. 
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In  order  to  show  the  error  space  more  clearly,  two  of  the  input 
variables  were  chosen  as  independent  variables  in  the  experiment:  the 
transit  time  for  pulses  (i.e.,  the  time  between  the  transmission  of  the 
pulse  and  the  receive  window) ,  and  the  number  of  search  pulses  in  the 
input  sequence.  Random  input  values  were  generated  to  simulate  a 
real-time  process.  The  number  of  assertion  violations  for  each  set  of 
input  values  was  tabulated  during  the  testing  for  use  in  constructing  a 
three-dimensional  grid. 

This  purpose  of  this  preliminary  experiment  was  to  determine  what 
the  error  space  of  a  program  looked  like  and  whether  it  was  feasible  to 
to  use  adaptive  search  techniques  to  maximize  an  error  function.  The 
results  of  this  experiment  showed  that  the  "error  surface"  from  testing 
this  set  of  modules  was  well  behaved.  It  contained  maximums,  minimums, 
and  gradients  which  could  be  used  by  various  search  techniques  to  locate 
the  values  in  the  input  space  which  cause  the  most  errors.  Figure  3 
shows  the  results  from  testing  this  program  with  the  set  of  input  values 
previously  illustrated  in  Figure  1. 
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Figure  3.  Example  of  an  Error  Space  Map 


A  second  experiment  on  a  much  larger  scale  is  now  in  progress.  A 

complex  software  program  which  computes  orbital  element  vectors  was 

selected  as  the  test  object.  Assertions  were  added,  and  a  set  of  errors 

for  "seeding"  the  program  were  generated  using  methods  developed  by 

current  research  .  The  errors  are  representative  of  those  found  in 
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large  programs  in  both  type  and  frequency  of  occurrence  ,  and  the  sites 
chosen  for  the  seeding  were  randomly  selected. 

The  input  variables  to  the  program  are  considered  two  at  a  time  to 
construct  a  three-dimensional  error  space.  Two  factors  should  be 
considered  in  deciding  which  variables  to  perturb:  it  is  most  effective 
to  choose  those  variables  which  have  the  most  influence  on  the  final 
output  values,  and  those  which  provide  collateral  testing  by  exercising 
other  input  variables  that  are  dependent  on  them. 

From  the  error  functions  derived  from  each  two-variable  case,  a 
complete  error  map  will  be  constructed  for  the  program.  The  error  space 
map  will  give  a  good  indication  of  the  shape  and  characteristics  of  the 
error  function  of  the  program.  The  shape  of  the  error  function  will 
determine  which  search  algorithms  are  most  applicable  to  locating  the 
input  values  of  the  program  which  lead  to  the  most  errors.  The 
performance  of  the  search  algorithm  chosen  for  the  experiment  will  be 
eval uated . 


The  final  results  of  the  current  experiment  will  provide  a  second 
evaluation  of  the  effectiveness  of  using  executable  assertions  for 
software  testing.  In  addition,  the  performance  of  the  adaptive  search 
technique  in  locating  the  maximum  value  of  the  error  function  will 
indicate  how  much  automation  is  possible  in  the  testing  of  computer 
programs . 
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Another  benefit  of  this  project  is  a  further  merging  of  the 
testing  process  with  the  requirements  of  fault  tolerance  in  software. 
Once  an  assertion  indicates  the  presence  of  an  error,  one  of  two  choices 
is  possible:  either  uncover  the  error  and  correct  the  code,  or  leave  the 
error  in  and  provide  a  method  to  recover  from  it.  Although  such  a 
suggestion  at  first  sounds  incomprehensible,  there  are  times  when  an 
error  cannot  be  found  or,  if  found,  is  so  complex  that  it  cannot  be 
corrected  easily  without  the  possibility  of  introducing  more  errors . 
Such  might  be  the  case  in  an  algorithm  that  is  basically  correct  but, 
within  certain  ranges,  produces  incorrect  results.  If  these  ranges  are 
not  expected  to  occur  often,  then  the  best  solution  may  be  to  provide 
compensatory  code  in  a  recovery  block. 

A  second  related  outcome  of  adaptive  testing  with  assertions  is 
that  the  tabulation  of  assertion  violations  indicates  which  assertions 
are  potentially  the  most  sensitive  to  errors  and  are,  therefore,  the 
most  valuable.  This  information  makes  it  possible  to  optimize  assertion 
coverage  in  fault  tolerant  applications  requiring  minimal  overhead. 

pi 
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executable  assertions,  it  is  possible  to  automatically ^execute  a  program^with  a 
large  number  of  tes^cases  over  a  wide  range  of  input  values.  The  usual  goal 
of  adaptive  testing  is  to  maximize  some  performance  value  (objective  function)-^ 
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Abstract  continued. 
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-p>  for  the  software  by  automated  perturbation  of  the  input  parameters  in  such 
a  way  as  to  degrade  the  system  performance  to  a  specified  limit/  ^his 
technique  only  indirectly  leads  to  locating  errors^-,  because  of  time- 
consuming  task  of  examining  the  usual  output  from  the  program. > In  software 
testing,  the  primary  goal  is  to  locate  the  maximum  number  of  errors,  rather 
than  maximize  the  performance  value.  Since  software  errors  can  "Be  1 
detected  by  executable  assertions,  these  assertions  can  be  used  to  define 
an  objective  function  for  the  adaptive  tester  so  that  a  program  can  be  test 
automatically  and  a  mapping  made  of  its  terror  space. ^  A  sej?sich  algorithm 
is  used  to  generate  new  test  cases  based  on  past  performance  data  about 
the  number  of  assertion  violations.  Software  testing  can  become  much  more 
efficient  and  effective  through  the-^Tise  of  adaptive  testing  with  assertions 
because  such  extensive  testing  increases  the  possibility  of  finding  any 
existing  errors  and  of  improving  software  reliability. 
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